Papers with image captioning systems
REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning (D19-1)
Copied to clipboard
| Challenge: | Existing metrics for image captioning evaluation provide an overall quality score, which is difficult to infer specific description errors. |
| Approach: | They propose a fine-grained evaluation method REO for automatically measuring the performance of image captioning systems. |
| Outcome: | The proposed method achieves higher consistency with human judgments and provides more intuitive evaluation results than other metrics. |
CapOnImage: Context-driven Dense-Captioning on Image (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning systems generate narrative captions for images, which are spatially detached from the image in presentation. |
| Approach: | They propose a task called captioning on image which generatesense captions at different locations of the image based on contextual information. |
| Outcome: | The proposed model achieves the best results in both captioning accuracy and diversity aspects. |
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Visual language grounding is widely studied in modern neural image captioning systems . a novel algorithm for crafting adversarial examples in image captions is proposed . |
| Approach: | They propose an algorithm to craft adversarial examples in machine vision and perception . their approach provides two evaluation approaches to check if they can mislead systems . |
| Outcome: | The proposed algorithm can craft visually-similar adversarial examples with randomly targeted captions or keywords, and the results are transferable to other image captioning systems. |
Women Wearing Lipstick: Measuring the Bias Between an Object and Its Related Gender (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches to visual understanding of image captioning rely on transformers and pre-trained paradigms to learn cross-modal representation. |
| Approach: | They propose a visual semantic-based gender score that measures the degree of bias and can be used as a plug-in for any image captioning system. |
| Outcome: | The proposed score can measure the bias relation between a caption and its related gender and can be used as an additional metric to the existing Object Gender Co-Occ approach. |